Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated accuracy distance metric to accomodate condition-wise approaches #1217

Merged
merged 2 commits into from
Sep 10, 2024

Conversation

ddcoggan
Copy link
Contributor

@ddcoggan ddcoggan commented Sep 5, 2024

Condition-wise accuracy distance metric as discussed with @benlonnqvist

@benlonnqvist
Copy link
Contributor

Thank you David!

PR #1213 depends on this.

@mike-ferguson
Copy link
Member

@benlonnqvist does this look good to you? If it does we can merge

@benlonnqvist
Copy link
Contributor

@mike-ferguson looks good to me! Thanks Mike & David.

@mike-ferguson
Copy link
Member

@benlonnqvist @ddcoggan I will force-merge this even though one test is not passing, as we are in the middle of modifying our testing suite. The important test here, default, has passed, so that is what matters.

@mike-ferguson mike-ferguson merged commit 4dd1d0f into brain-score:master Sep 10, 2024
11 of 12 checks passed
KartikP added a commit that referenced this pull request Sep 11, 2024
* Updated accuracy distance metric to accomodate condition-wise approaches (#1217)

Co-authored-by: ddcoggan <[email protected]>

* Add temporal metrics; add temporal versions of MajajHong2015 (#1109)

* feature: support temporal models for neural alignment by chaning TemporalIgnore to Temporal Aligned

* add example temporal submission

* complete new framework

* new module: temporal model helpers

* change the arch of temporal; add tutorials

* improve: better naming

* update: wrapper tutorial on brain model

* add feature: inferencer identifier tracked by extractor for result caching

* fix: video fps sampling; need more tests!

* fix bugs: video sampling based on fps was wrong.

* add mmaction2 models; add more features to the inferencers

* PR: temporal model helpers

* PR fix: not including gitmodules for now

* Update brainscore_vision/model_helpers/brain_transformation/temporal.py

Co-authored-by: Martin Schrimpf <[email protected]>

* Update brainscore_vision/model_helpers/brain_transformation/temporal.py

Co-authored-by: Martin Schrimpf <[email protected]>

* Update brainscore_vision/model_helpers/brain_transformation/temporal.py

Co-authored-by: Martin Schrimpf <[email protected]>

* Update brainscore_vision/models/temporal_models/test.py

Co-authored-by: Martin Schrimpf <[email protected]>

* add mae_st; add ding2012

* try new arch

* init ding2012

* add tests for temporal model helpers; add block inferencer

* Delete tests/test_model_helpers/temporal/test___init__.py

delete the old test

* add benchmark ding2012

* add mutliple libs for temporal models

* change executor output format; add more inference tests; init load_weight in s3

* add openstl

* update backend for executor

* feat:load_weight_file and corresponding test

* change:resize strategy changed from bilinear to pooling

* change:resize strategy changed from bilinear to pooling

* fix mae_st submission

* minor

* fix:dtype in assembly time align

* minor

* update model submissions

* fix dependency

* refactor: simplify the inferencer methods

* fix:block inferencer, neuroid coord while merging

* fix:inferencer identifier

* fix:weigh download

* change tests to have max_workers=1

* revert screen.py

* not submit region_layer_map

* remove torch dependency

* make fake modules in tests

* add torch to requirements; avoid torch in tests

* minor

* minor

* np.object changed to object

* remove return in tests

* fix insertion position bug

* Apply suggestions from code review

add: more type hints

Co-authored-by: Martin Schrimpf <[email protected]>

* add: more type hints and comments

* minor

* pr:only commit temporal model helpers

* pr: add one model for example

* undo whole_brain in Brainodel.RecordingTarget

* use logger and fix newlines

* fix: video fps with copy was wrong

* feat:fractional max_spatial_size

* downsample layers in VideoMAE

* fix:video sampling wrong duration

* add more tests

* fix merge

* fix merge

* module refactor; add more input test

* add more temporal models

* fix videomaev2 sha

* fix:temporal_modelmae_st

* change:video conservative loading; rename:image to pil image

* fix:video last frame sampling; fix_time_naming

* ignore pytest_cache

* re-trigger tests

* add joblib pool error management; fix video/image path recognizer

* update: naming of failed to pickle func in joblibmapper

* add temporal metric helpers

* add temporal version of mamjajhong2015

* Update benchmark.py

type hint

* Update benchmark.py

* Update brainscore_vision/metric_helpers/temporal.py

Co-authored-by: Martin Schrimpf <[email protected]>

* Update brainscore_vision/metrics/internal_consistency/__init__.py

Co-authored-by: Martin Schrimpf <[email protected]>

* Update benchmark.py

---------

Co-authored-by: Yingtian Tang <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: deirdre-k <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>

* Python 3.11 Migration (v2.1.0) (#1199)

* changes necessary for 3.11

* change references of 3.7 to 3.11, remove tf and keras

* fix geirhos data tests

* Normalize deprecated

* np.float is deprecated

* update w/ master (#1066)

* add r101_eBarlow_lmda_01_1 to models (#1037)

Co-authored-by: AutoJenkins <[email protected]>

* update hmax requirements.txt (#1053)

* update hmax requirements.txt to add torchvision (#1054)

* fix mobilenet `pooler` layer prefix (#1055)

* add r34_eMMCR_Mom_lmda_02_1 to models (#1061)

Co-authored-by: AutoJenkins <[email protected]>

* add r101_eBarlow_lmda_02_1_copy to models (#1059)

Co-authored-by: AutoJenkins <[email protected]>

* add r34_eMMCR_Mom_lmda_01_1 to models (#1060)

Co-authored-by: AutoJenkins <[email protected]>

* add r34_eMMCR_Mom_Vanilla_1 to models (#1063)

Co-authored-by: AutoJenkins <[email protected]>

* ignore time dimension in Bracci2019 (#1064)

---------

Co-authored-by: Katherine Fairchild <[email protected]>
Co-authored-by: AutoJenkins <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>

* update (#1093)

* add r101_eBarlow_lmda_01_1 to models (#1037)

Co-authored-by: AutoJenkins <[email protected]>

* update hmax requirements.txt (#1053)

* update hmax requirements.txt to add torchvision (#1054)

* fix mobilenet `pooler` layer prefix (#1055)

* add r34_eMMCR_Mom_lmda_02_1 to models (#1061)

Co-authored-by: AutoJenkins <[email protected]>

* add r101_eBarlow_lmda_02_1_copy to models (#1059)

Co-authored-by: AutoJenkins <[email protected]>

* add r34_eMMCR_Mom_lmda_01_1 to models (#1060)

Co-authored-by: AutoJenkins <[email protected]>

* add r34_eMMCR_Mom_Vanilla_1 to models (#1063)

Co-authored-by: AutoJenkins <[email protected]>

* ignore time dimension in Bracci2019 (#1064)

* Update behavior.py (#742)

Co-authored-by: Sam Winebrake <[email protected]>
Co-authored-by: deirdre-k <[email protected]>

* add eBarlow_lmda_02_1_full to models (#1067)

Co-authored-by: AutoJenkins <[email protected]>

* oddoneout: build full stimulus id index for non-numeric ids (#1068)

* Add mobilevit_small - take 2 (#1051)

* Add mobilevit_small - take 2

* Update brainscore_vision/models/mobilevit_small/model.py

Co-authored-by: Martin Schrimpf <[email protected]>

---------

Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>

* add yudixie_resnet18_240719_1 to models (#1070)

Co-authored-by: AutoJenkins <[email protected]>

* add yudixie_resnet18_240719_0 to models (#1069)

Co-authored-by: AutoJenkins <[email protected]>

* add yudixie_resnet18_240719_2 to models (#1071)

Co-authored-by: AutoJenkins <[email protected]>

* add yudixie_resnet18_240719_10 to models (#1079)

Co-authored-by: AutoJenkins <[email protected]>

---------

Co-authored-by: Katherine Fairchild <[email protected]>
Co-authored-by: AutoJenkins <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: Linus Sommer <[email protected]>
Co-authored-by: deirdre-k <[email protected]>
Co-authored-by: Kartik Pradeepan <[email protected]>

* remove keraswrapper

* remove brendel

* Trying a fix for travis

* add necessary imports

* Updating ubuntu version as xenial doesn't have 3.11

* Adding fix for pyproject.toml

* Pin numpy below 2.0

* pointing to test branch for travis

* update s3 bucket to brainscore-unittests

* update s3 bucket to brainscore-unittests (rajalingham2020)

* update s3 bucket to brainscore-unittests (sanghavi2020)

* update folder name of models to folder_name="models-to-integrate-for-2.0"

* update folder name of MAE model to folder_name="models-to-integrate-for-2.0"

* update folder name of MAEv2 to folder_name="models-to-integrate-for-2.0"

* update folder name of mae_st to folder_name="models-to-integrate-for-2.0"

* PredNet commented out in init

* remove prednet dead code

* remove prednet code

* scialom data: update image_id to stimulus_id

* add number_of_trials keyword to rajalingham2018

* pointing to branch

* Removing dash from python module name (#1118)

* Removing dash from module name (#1119)

* add require_variance keyword to precomputedfeatures call

* add s3 download if not present to test helper

* add missing s3 import

* Update test_helper.py

* pull item from ceiling

Need to check with martin if this will break anything

* update coggan benchmark to work w/ new pandas/python

* add identifier function to pre computed features

* forgot to add string line

* update rajalingham2018 deprecated test

* update w/ master (#1194)

* add r101_eBarlow_lmda_01_1 to models (#1037)

Co-authored-by: AutoJenkins <[email protected]>

* update hmax requirements.txt (#1053)

* update hmax requirements.txt to add torchvision (#1054)

* fix mobilenet `pooler` layer prefix (#1055)

* add r34_eMMCR_Mom_lmda_02_1 to models (#1061)

Co-authored-by: AutoJenkins <[email protected]>

* add r101_eBarlow_lmda_02_1_copy to models (#1059)

Co-authored-by: AutoJenkins <[email protected]>

* add r34_eMMCR_Mom_lmda_01_1 to models (#1060)

Co-authored-by: AutoJenkins <[email protected]>

* add r34_eMMCR_Mom_Vanilla_1 to models (#1063)

Co-authored-by: AutoJenkins <[email protected]>

* ignore time dimension in Bracci2019 (#1064)

* Update behavior.py (#742)

Co-authored-by: Sam Winebrake <[email protected]>
Co-authored-by: deirdre-k <[email protected]>

* add eBarlow_lmda_02_1_full to models (#1067)

Co-authored-by: AutoJenkins <[email protected]>

* oddoneout: build full stimulus id index for non-numeric ids (#1068)

* Add mobilevit_small - take 2 (#1051)

* Add mobilevit_small - take 2

* Update brainscore_vision/models/mobilevit_small/model.py

Co-authored-by: Martin Schrimpf <[email protected]>

---------

Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>

* add yudixie_resnet18_240719_1 to models (#1070)

Co-authored-by: AutoJenkins <[email protected]>

* add yudixie_resnet18_240719_0 to models (#1069)

Co-authored-by: AutoJenkins <[email protected]>

* add yudixie_resnet18_240719_2 to models (#1071)

Co-authored-by: AutoJenkins <[email protected]>

* add yudixie_resnet18_240719_10 to models (#1079)

Co-authored-by: AutoJenkins <[email protected]>

* Add cv_18_dagger_408_pretrained (#1104)

* Add model files

* Adjust timm version

---------

Co-authored-by: Ethan Pellegrini <[email protected]>

* add eBarlow_lmda_02_200_full to models (#1121)

Co-authored-by: AutoJenkins <[email protected]>

* remove old tutorials in favor of new link (#1170)

---------

Co-authored-by: Katherine Fairchild <[email protected]>
Co-authored-by: AutoJenkins <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: Linus Sommer <[email protected]>
Co-authored-by: deirdre-k <[email protected]>
Co-authored-by: Kartik Pradeepan <[email protected]>
Co-authored-by: pellegreene <[email protected]>
Co-authored-by: Ethan Pellegrini <[email protected]>

* environment lock for 2.1.0

* Rename environment.yml to environment_lock.yml

* add test dependencies and move certain dependencies to pip

* remove branch from core dependency

* new env lock with all dependencies of env (including test)

* add informative comment to env lock

* Update .travis.yml

* point to 3.11 branches

* update pointers to 3.11 branches

* add back check for forks in travis

* remove f string from full_name variable

* remove if/else that enabled tf

* remove folder_name for models that were moved to correct directory

* version less specific (2.1.0 to 2.1)

* remove protobuf

* remove python specification in setup.pys

* Update mismatched s3 folder name

* Update environment_lock.yml w/ main branches

* update branch pointers to main

---------

Co-authored-by: Katherine Fairchild <[email protected]>
Co-authored-by: AutoJenkins <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: Linus Sommer <[email protected]>
Co-authored-by: deirdre-k <[email protected]>
Co-authored-by: Kartik Pradeepan <[email protected]>
Co-authored-by: Deirdre Kelliher <[email protected]>
Co-authored-by: pellegreene <[email protected]>
Co-authored-by: Ethan Pellegrini <[email protected]>

---------

Co-authored-by: David Coggan <[email protected]>
Co-authored-by: ddcoggan <[email protected]>
Co-authored-by: YingtianDt <[email protected]>
Co-authored-by: Yingtian Tang <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: Martin Schrimpf <[email protected]>
Co-authored-by: deirdre-k <[email protected]>
Co-authored-by: Michael Ferguson <[email protected]>
Co-authored-by: Sam Winebrake <[email protected]>
Co-authored-by: Katherine Fairchild <[email protected]>
Co-authored-by: AutoJenkins <[email protected]>
Co-authored-by: Linus Sommer <[email protected]>
Co-authored-by: Deirdre Kelliher <[email protected]>
Co-authored-by: pellegreene <[email protected]>
Co-authored-by: Ethan Pellegrini <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants